The codes can be run using MATLAB. 
Run RegretAnalysis.m file. 
Change the system parameters as you need

ActionValue.m : Calculates action value function with predicted transition kernels 
OurPolicy.m: Our Policy
Uniform.m: random policy
UCWhittle.m: UCWhittle Policy
WhittleOracle.m: Whittle Index Policy with with True Transition Kernels
WhittleIndex.m: Calculates Whittle index with predicted transition kernels
WIQL; WIQL policy
AoIfunction.m: reward calculator
WhittleIndexUC.m: Calculates Whittle index with predicted transition kernels for UCWhittle Policy